00100	II.  RESEARCH PLAN
00200	
00300		A.  THE PILOT ASSEMBLY STATION
00400	
00500	The basic goal of the proposed project  is  to  implement  a  general
00600	purpose  assembly  station  and  to  prove  its applicability to real
00700	problems.  The project will be considered a success if  one  or  more
00800	enterprises  chooses  to  manufacture  and market the general purpose
00900	station.  There are a number of hardware and software problems to  be
01000	solved in meeting our goal.
01100	
01200	In one configuration, the assembly station is assumed to be connected
01300	to a large time-shared computer facility. Not only does  this  appear
01400	to  be a good design for our problem, but it is also necessary for an
01500	integrated  programmed  automation  facility.   The  only  emphasized
01600	statement in the ARPA study report [Anderson 1972] is:   "The maximum
01700	impact  of  computers  in  manufacturing  will  come  from  complete,
01800	real-time  computer  cognizance  and  control  of  all  processes and
01900	resources, allowing the precise scheduling and  allocation  of  these
02000	resources."
02100	
02200	Each  assembly  station will consist of a small digital computer, one
02300	or more manipulators and cameras. Manipulators and  cameras  will  be
02400	combined  as  modules  in  systems  to suit a given task, and will be
02500	digitally controlled by the station computer.  The  station  computer
02600	may  be  connected  to  a  central  computer or may stand alone.  The
02700	latter configuration is less efficient in the long  run,  but  allows
02800	for incremental automation.
02900	
03000	The program in the station computer will be flexible as to the number
03100	and position of manipulators and cameras. It will control input  from
03200	the  cameras,  provide for some pre-processing of the picture and the
03300	application of simple tests to the visual data.     In  the  case  of
03400	manipulators  it will provide for simultaneous servo control. It will
03500	also process  other  inputs,  such  as  from  touch  sensors  on  the
03600	manipulators,  from photocell interrupt devices, limit switches, etc.
03700	Both analog and binary outputs will be available for the  control  of
03800	other devices.
03900	
04000	Each  station  computer will have access to a task program specifying
04100	the actions and tests it should perform to carry out its task.   This
04200	task program will be compiled by the central computer.    Compilation
04300	will consist of the analysis of predicted scenes in order to  specify
04400	simple tests that the station computer can carry out, the calculation
04500	of collision free trajectories for the manipulators, the sequence  of
04600	operations  and  simple  contingency plans. etc.  We are planning for
04700	tasks  whose  normal  execution  does  not  require  detailed   scene
04800	analysis.   If scene analysis becomes necessary (e.g. a dropped part)
04900	we envision the station sending an image to the central computer  for
05000	processing.
05100	
05200	The  central  computer will perform overall control of the individual
05300	assembly stations and the relative rates of production.   If  at  any
05400	station  a  planned test fails, for which a contingency plan does not
05500	exist, or any unexpected sensor signals  are  received,  the  central
05600	computer  will  also be interrupted and appropriate actions taken. If
05700	the assembly station is not connected to a  central  computer,  these
05800	operations will require human intervention.
05900	
06000	
06100	Most of the hardware required  for  the  assembly  station  has  been
06200	developed  for  the  laboratory.   Manipulators  and  cameras will be
06300	discussed in detail below. The major additional problem is to  refine
06400	the   existing   designs   to  meet  the  demands  of  an  industrial
06500	environment.
06600	
06700	We are now acquiring small computer  such  as  the  PDP-11  and  will
06800	interface  the existing arms to it together with one or more cameras.
06900	A software package for the small computer will then be  developed  in
07000	order to control the manipulators, read the TV cameras and to perform
07100	low level visual analysis.  Programs to  plan  tasks,  calibrate  the
07200	system  and  to  perform higher level visual analysis, error recovery
07300	and strategy planning will be written in a high  level  language  for
07400	the time-sharing processor.
07500	
07600	The most difficult part of the assembly station, and the heart of the
07700	project is a demonstration of the feasibility of the station. We plan
07800	to do this by developing programs which actually assemble and inspect
07900	a variety of fairly complex manufactured items.
08000	
08100	A typical example is our current effort, assembly  of  an  automobile
08200	water  pump.  The  major  parts  are  to  be  located, the screws are
08300	supplied by a feeder, as is standard.  Locating the body and cover of
08400	the  pump  require  a gross localization, then a fine localization of
08500	screw holes.    Touch  provides  a  final  accuracy  and  independent
08600	coordination  of  the  hand.    The  placing of alignment pins within
08700	screw holes could be monitored visually; although  in  this  case,the
08800	previous  actions  provide  adequate accuracy.   Placing of the cover
08900	over alignment pins can be  carried  out  without  vision.   Improved
09000	force   sensing  now  under  development  will  be  useful  for  that
09100	operation.
09200	
09300	As we move to more difficult tasks, the sophistication required  goes
09400	up  quite  rapidly.   The  current  manipulation  routines seem to be
09500	expandable to cover anticipated problems, especially since we  intend
09600	to  employ special purpose manipulators (tools) for some tasks.   The
09700	main part of our effort will be vision for assembly; an assembly task
09800	will  be performed using vision, touch, and two hands.     The vision
09900	effort will be integrated with other abilities, and  while  no  great
10000	generality  will  be  stressed, new modules will be incorporated into
10100	the system, and a set of techniques will evolve.  These  will  depend
10200	heavily  on  techniques  similar  to  those  of our recent studies in
10300	visual feedback and hand/eye coordination [Gill 1972].  The  internal
10400	model  of  the  world  will  be  used to predict visual appearance of
10500	selected areas of the scene, and a  set  of  verification  procedures
10600	will  confirm  or contradict the predictions.  Many of the operations
10700	will be  connected  with  supplying  feedback  to  manipulation,  for
10800	alignment and insertion. These visual procedures will have models for
10900	the parts and approximate locations.
11000	
11100	This and other visual tasks connected with assembly will initially be
11200	performed  as  special  case  problems.    In addition, to make these
11300	techniques  available,  we  will  pursue  development   of   strategy
11400	programs.  This step is essential if intelligent assembly devices are
11500	to be conveniently programmed for complex tasks. Within the  division
11600	of  labor between large and small machines, this facility must reside
11700	in the central time-sharing facility which  has  large  memory,  file
11800	storage,  and problem-solving languages. The system must have a model
11900	of  the  parts  for  assembly,and  a  sequence  of  instructions  for
12000	assembly.  It will be necessary to have a representation of shape and
12100	models for usual assembly operations such as insertion and alignment,
12200	and  the  ability  to  solve simple problems in connection with these
12300	operations, all this to  enable  convenient  communication  with  the
12400	user.  When  these facilities exist, the addition of new features and
12500	operations can be incorporated smoothly, so that users can call on  a
12600	growing library of planning modules.  The result of a planning effort
12700	is a cycle of machine  planning  and  man-machine  interaction  which
12800	results  in  a  plan  network  for  the  mini computer which controls
12900	individual machines.
13000	
     

00100		B. RESEARCH IN VISION
00200	Research in computer vision has progressed so that the larger part of
00300	our effort is directed toward vision in real scenes. We outline past
00400	research, including work on visual feedback, development of useful
00500	modules, analysis of some outdoor scenes, interpretation of scenes
00600	with curved objects, and a study of sensor systems.
00700	
00800	Our past work on visual feedback [Gill 1972] in stacking blocks and
00900	inserting objects in holes is directly relevant to vision for
01000	assembly. The issues involved are: coordinating hand and eye,
01100	utilizing incremental accuracy of vision and manipulation, accurate
01200	prediction to limit search, and limiting accumulation of errors.
01300	Visual guidance in picking up objects and stacking or inserting
01400	objects overcomes errors in the visual and arm coordinate systems,
01500	and errors in correspondence between the two coordinate systems. This
01600	was done under realistic conditions, with the hand still holding the
01700	object and in view; the system predicts the appearance of edges and
01800	corners within a small window around the edges to be aligned, and
01900	uses a corner/edge finder to determine alignment error. The arm
02000	makes accurate incremental error correction. If predictions can be
02100	made accurately, the search window can be small, which increases the
02200	speed of response and decreases possibility of confusion with other
02300	areas of the scene. A major limitation to accuracy of prediction
02400	(and to long-term reliability in the industrial environment) is drift
02500	of calibration. That work incorporates calibration updating by
02600	self-calibration to coordinate hand and eye coordinate systems and
02700	provide for reliability, as well as to limit search by accurately
02800	predicting expected features. Visual feedback with stereo vision
02900	is much more powerful than without; we have demonstrated stereo
03000	visual feedback in a program [Binford unpublished].
03100	
03200	We have a set of visual modules to be used in a goal-directed
03300	way. These modules answer questions which arise in practical visual
03400	feedback tasks, and in analysis of outdoor scenes. Are there color
03500	edges in a selected window? Are there edges in brightness? What is
03600	the color of the region in the window?
03700	Several
03800	preliminary color region analysis programs have been developed
03900	[Binford, unpublished, Bajcsy 1972]. An edge operator [Hueckel 1969]
04000	and a region-based corner/edge finder [Gill 1972] both operate on
04100	small windows of the scene, as directed by a higher level. An edge
04200	verifier is used to confirm or reject edges predicted from context
04300	[Tenenbaum 1970]. Color identification has long been demonstrated.
04400	Recently, a program has been implemented [Thomas unpublished]
04500	 based on Land's Retinex
04600	model of color constancy in vision, which defines a color triangle
04700	from ordering of brightnesses. The center of the triangle is
04800	designated as white. The program shows a color constancy similar to
04900	humans; table top scenes are assigned colors reasonably well under a
05000	variety of lighting with different spectra: sunlight, flourescent,
05100	and incandescent.
05200	
05300	Understanding outdoor scenes is a more complex task than many industrial
05400	visual operations, but it has utility for planetary exploration; it is
05500	also a useful exercise for developing color, texture, and stereo operations.
05600	Bajcsy analyzed outdoor scenes with a combination of color
05700	and texture region analysis [Bajcsy 1972]; depth cues were taken from
05800	the gradient of texture.
05900	An attempt was made to obtain consistent interpretations by making
06000	three dimensional interpretations, and confirming boundaries by
06100	independent properties.
06200	 Natural objects and natural scenes have been
06300	used for our work with motion parallax [Nevatia in draft] and stereo
06400	vision [Quam,Hannah].These are complementary techniques of depth
06500	perception which are applicable for planetary exploration and
06600	industrial tasks. Motion parallax can be used for planning of
06700	navigation and characterizing large scale terrain features, since it
06800	provides large baselines.
06900	
06905	Another aspect of vision with direct application to industrial
06910	enviroments is explored in our work with curved objects. Current
06915	programs use depth information to make useful descriptions of objects like a doll,
06920	a toy horse, a snake, or a glove [Agin]. Programs can segment
06925	these objects into parts like legs, arms or fingers, and link
06930	these parts together to form descriptions of entire objects [Nevatia].
06935	These operations are precursors to a recognition process. The
06940	work depends on a new representation of shape [Binford 1971] in terms
06945	of volume components and a part/whole segmentation. Primitive parts
06950	are described by an axis which is a space curve, and normal cross
06955	sections along the axis. The representation is based on a formalism,
06960	"generalized translational invariance". There are significant theoretical
06965	reasons why this representation leads to a natural choice of parts
06970	by simple procedures, in distinction to other suggested representations.
07100	We developed a laser ranging device [Binford
07200	1970,Agin] which was operational two years ago.
07205	It is particularly interesting that the device adds a cost of only
07210	about $400 to the cost of a visual system.
08000	
08100	A study of visual and depth ranging sensor has been carried out
08200	which provides a background for choosing sensors and interface, and
08300	making cost estimates [Binford 1972, Earnest 1967].  We have
08400	maintained a calibrated hand and vision system for several years
08500	[Sobel 1970], and have developed self-calibration techniques with one
08600	and two eyes [Yakimovsky unpublished, Hannah 1972, Gill 1972].
08700	
08800	A system for understanding scenes of blocks from single views has
08900	been implemented [Falk 1970]; it segments objects, matches against
09000	internal models, predicts appearance from its assumptions, and
09100	searches for verification and contradiction based on the predicted
09200	image. Support and spatial relations are also obtained. There has
09300	been past work on programs to obtain line drawings as input to
09400	recognition systems. An earlier system was based on limited context
09500	of expecting continuation of edges at vertices and predicting missing
09600	edges in certain expected configurations [Grape 1970]. A more recent
09700	approach has been to use models within the process of generating line
09800	drawings, so that the most efficient next guess can be made and
09900	context used intrinsically in the formation of line drawings.
10000	
10100	VISION FOR ASSEMBLY
10200	
10205	This is a key part of the proposed project. We propose to evaluate and
10210	implement visual facilities required by a robot with simple vision,
10215	in conjunction with force and touch. These facilities must be
10220	modularized by working out several examples, and made fast enough and small 
10225	enough to be practical for a minicomputer-based robot. We have considerable
10230	experience with such standardization, and with visual feedback techniques
10235	in the domain of blocks. We are now at work on extending visual feedback
10240	techniques to typical assembly tasks. We are also evaluating methods
10245	of automating visual feedback. We have well-developed systems for
10250	modeling objects from the console and to predict their appearance using 
10255	techniques from computer graphics [Baumgart 1972a,b]. We can predict
10260	curved edges from our models. We will use simple region analysis programs and edge operators
10265	to locate curved edges predicted within a small window. Automated visual
10270	feedback would combine with evolving strategy programs in an interactive
10275	system, Assembly Robot Management System, ARMS. We will maintain a calibrated system and
10285	improve calibration updating so that reliability is maintained at a
10290	high level.
10295	
10300	We have developed a special case program for locating the pump body 
10305	in the pump assembly and moving it to the standard location. We are
10310	at work on several other such tasks involved in the pump assembly
10315	which contain components of general utility for visual feedback.
10320	These tasks are typical of the simple visual operations which we
10325	will implement or adapt.
11800	
11900	VISION RESEARCH
12000	
12005	While continuing research activities are not to be supported by the
12010	proposed project, in all likelihood, the research will suggest
12015	useful programs for the assembly system, particularly work in 
12020	recognition of complex objects using depth data.  The computer now
12025	has symbolic internal models of objects.  We are now in a position
12030	to recognize objects such as a glove, a hammer, etc.  We will
12035	immediately deal with matching descriptions for recognition.  The
12040	important extensions we wish to deal with are occlusion and access
12045	to a large visual memory.  These will involve matching of partial
12050	descriptions and encoding of partial descriptions as indices
12055	to a small class of similar shapes, which can then be compared in more
12060	detail.  We are working toward making the hardware stable enough
12065	that it can maintain calibration, and software streamlined so that
12070	the depth measurement system can be integrated with other hand/eye
12075	facilities.
13705	 We intend to apply
13800	large parts of our programs for description and recognition of curved
13900	objects using only monocular intensity information. This is a much
14000	harder problem, but within our representation, shares much of the
14100	same code, and relies on the same conceptual basis of local
14200	three-dimensional models.
14300	
15000	
15650	 There is a growing effort on navigation of vehicles in outdoor
15700	scenes. While this is not directly relevant to the industrial
15800	environment, it does reflect the growing involvement in complex
15900	visual tasks, and will lead to further development of an integrated
16000	visual system.
16005	One approach which has already given initial results [Yakimovsky]
16010	is to combine region growing with interpretation using a world
16015	model.  Thus, regions are combined not simply on the basis of
16020	uniformity but because combining them improves the interpretation.
16025	At the same time, we will be combining more facilities into these
16030	visual systems, and more usable knowledge in the world model.
16100